Augmented reality display of scene behind surface

ABSTRACT

Embodiments are disclosed that relate to augmenting an appearance of a surface via a see-through display device. For example, one disclosed embodiment provides, on a computing device comprising a see-through display device, a method of augmenting an appearance of a surface. The method includes acquiring, via an outward-facing image sensor, image data of a first scene viewable through the display. The method further includes recognizing a surface viewable through the display based on the image data and, in response to recognizing the surface, acquiring a representation of a second scene comprising one or more of a scene located physically behind the surface viewable through the display and a scene located behind a surface contextually related to the surface viewable through the display. The method further includes displaying the representation via the see-through display.

BACKGROUND

Surfaces, such as walls and doors, may obscure a scene from view. Inorder to view the scene, a person may need to open, or otherwisemanipulate, the surface while in close physical proximity to thesurface. However, such manipulation may not be possible or desirable incertain situations, such as when the surface is out of reach.

SUMMARY

Embodiments are disclosed that relate to augmenting an appearance of asurface via a see-through display device. For example, one disclosedembodiment provides, on a computing device comprising a see-throughdisplay device, a method of augmenting an appearance of a surface. Themethod comprises acquiring, via an outward-facing image sensor, imagedata of a first scene viewable through the display. The method furthercomprises recognizing a surface viewable through the display based onthe image data and, in response to recognizing the surface, acquiring arepresentation of a second scene comprising one or more of a scenelocated physically behind the surface viewable through the display and ascene located behind a surface contextually related to the surfaceviewable through the display. The method further comprises displayingthe representation via the see-through display.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example use environment for an embodiment of asee-through display device, and also illustrates an embodiment of anaugmentation of a view of a scene by the see-through display device.

FIGS. 2 and 3 illustrates other embodiments of augmentations of a viewof a scene by the see-through display device of FIG. 1.

FIG. 4 schematically shows a block diagram illustrating an embodiment ofa use environment for a see-through display device.

FIG. 5 shows a process flow depicting an embodiment of a method foraugmenting a view of a scene.

FIG. 6 schematically shows an example embodiment of a computing system.

DETAILED DESCRIPTION

As mentioned above, various surfaces may obscure a person's view ofscenes located behind the surfaces. In some instances, it may beadvantageous for the person to have the ability to view what is locatedbehind the surface without gaining an actual, physical view behind thesurface. For example, in the case of a user-manipulable surface such asa refrigerator door, manipulating the surface to gain a view behind thesurface may allow cold air to escape. Likewise, it may be desirable forthe sake of convenience to view behind a surface when not in physicalproximity to the surface, such as when a person is silting on a sofaacross the room from the surface, or is in a different location from thesurface.

Accordingly, embodiments are disclosed that relate to providing a visualrepresentation of an obscured scene, for example, via displayingrepresentation of the obscured scene in spatial registration with anobscuring surface or contextually related surface. In this way, a usermay be able to visually interpret an obscured scene even if the user hasnot previously viewed the obscured scene and/or is not in spatialproximity to the obscured scene.

Prior to discussing these embodiments in detail, a non-limiting usescenario is described with reference to FIG. 1, which illustrates anexample environment 100 in the form of a kitchen. The kitchen comprisesa scene 102 viewable through see-through display device 104 worn by user106. It will be appreciated that, in some embodiments, the scene 102viewable through the see-through display may be substantiallycoextensive with the user's field of vision, while in other embodimentsthe scene viewable through the see-through display may occupy a portionof the user's field of vision.

As will be described in greater detail below, see-through display device104 may comprise one or more outwardly facing image sensors (e.g.,two-dimensional cameras and/or depth cameras) configured to acquireimage data (e.g. color/grayscale images, depth images/point cloud data,etc.) representing environment 100 as the user navigates theenvironment. This image data may be used to obtain information regardingthe layout of the environment (e.g., three-dimensional surface map,etc.) and of objects and surfaces contained therein.

The image data acquired via the outwardly facing image sensors may beused to recognize a user's location and orientation within the room. Forexample, one or more feature points in the room may be recognized bycomparison to one or more previously-acquired images to determine theorientation and/or location of see-through display device 104 in theroom.

The image data may be further used to recognize a surface, such assurface 108 (e.g., refrigerator door), that obscures another scene.Recognition of a surface may comprise, for example, detecting an openingand/or closing of the surface via the image data, detecting a shape of adoor or such feature in the image data, etc. As another example,see-through display device 104 may determine the existence of image datafor a scene located behind a detected surface, and may thereforerecognize the surface as obscuring a scene behind the surface withoutdirectly detecting an opening/closing motion of a door, withoutclassifying the appearance of the object comprising the surface, etc.Further, in some embodiments, see-through display device 104 may beconfigured to determine a context of scene 102 (e.g., refrigerator,living room, office, closet, etc.) and/or of a surface (e.g.,refrigerator door, cabinet door, wall, etc.) viewable through thedisplay device. Such contexts may be usable, for example, toprogrammatically determine whether to display image data of a scenebehind the surface (e.g. based upon one or more user preferences). Asnon-limiting examples, a user may desire to view image data for scenesobscured by a door, for scenes located within their home, for interiorrefrigerator scenes, and/or for scenes comprising any other suitablecontext. Accordingly, upon recognizing one or more scenes comprisingsuch contexts, representations of said scenes may be programmaticallydisplayed. Such contexts may be further usable, for example, todetermine whether to display image data of a scene behind the surfacebased upon privacy preferences, and, if such display is permissible, todetermine which data to display (e.g. what surface “depth” in instanceswhere one recognized surface is located behind another recognizedsurface, whether to display a most recent or older image of a scene,etc.). Therefore, such contexts may allow for scene-based and/orsurface-based granularity regarding the sharing, selection, and displayof various scenes.

See-through display device 104 is further configured to augment theappearance of surface 108 by displaying representation 110 (e.g., imagedata) of scene 112 (e.g., refrigerator interior) located physicallybehind surface 108 as an “overlay” on top of surface 108 (i.e.,refrigerator door). As will be described in greater detail below, suchaugmentation may be triggered via any suitable mechanism, including, butnot limited to, user command and/or surface recognition by the displaydevice. As another example, in some embodiments, see-through displaydevice 104 may be configured to determine a direction of a gaze of user106 (e.g., via one or more imaging sensors imaging a position of one orboth of the user's eyes), and representation 110 may be triggered basedupon the user's gaze at surface 108.

Representation 110 of scene 112 may comprise previously-collected imagedata. For example, the representation may comprise image data thatpreviously collected by see-through display device 104 during a previousinteraction of user 106 with the object incorporating surface 108. Asanother example, the displayed representation may comprise image datathat was previously-collected by a different device (e.g., see-throughdisplay device of another user, smartphone, IP camera, etc.).Accordingly, in some embodiments, see-through display device 104 may beconfigured to share data with, and retrieve data from, multiple devicesin order to provide a most recently-acquired image. Further, in yetother embodiments, a user may choose to view an older representationthan a most-recently acquired image, as explained in more detail below.

It will be appreciated that the displayed representation of the obscuredscene may comprise information generated from the image data instead of,or in addition to, the image data itself. For example, in someembodiments, the representation may comprise a generated model (e.g.generated from point cloud data acquired via a depth camera) and/orgenerated textual description of scene 112. In some embodiments, aviewed angle/orientation of such a generated model may be varied by theuser.

Although representation 110 of scene 112 is illustrated as being inspatial registration and coextensive with the portion of surface 108viewable through the see-through display, it will be appreciated thatrepresentation 110 may be displayed in any other suitable manner, andmay be displayed via any other suitable device than a see-throughdisplay device. For example, in some embodiments, augmentation of scene102 may be provided via a mobile computing device that does not includea see-through display, as mentioned above. In such an embodiment, ascene may be imaged via an image sensor of a mobile phone, tabletcomputer, or other mobile device, and a representation of scene 102(e.g., a “live feed” from image sensor) may be displayed in conjunctionwith representation 110, for example, as an overlay over surface 108.

As yet another example, FIG. 2 illustrates an example embodiment ofscene 200 within environment 202 as viewed through a see-through displaydevice (e.g., see-through display device 104 of FIG. 1). As illustrated,environment 202 is shown in the form of a grocery store and comprisessurface 204 (e.g., see-through door) of object 206 in the form of arefrigerated display case.

The see-through display device may be configured to identify that object206 is a refrigerated display case, and further to determine that object206 is contextually related to another object, such as the refrigeratorcomprising surface 108 of FIG. 1. Such a determination may be made basedupon analysis of a shape and/or appearance of the object againstrecognized shapes (e.g. via classification function(s)), based upon ashape and/or appearance of contents of the object (e.g. milk cartons orin any other suitable manner. Further, additional contextual informationmay be taken into account when identifying an object. For example,locational information (e.g. that the user is in a grocery store) may beused to help identify object 206.

In response to identifying object 206, the see-through display devicedisplay an image augmenting the appearance of surface 204, wherein theimage comprises representation 208 of contextually-related scene 210(refrigerator interior scene 112 of FIG. 1 in this example). In thismanner, the contextual clue of the refrigerated display case in agrocery store, and/or the contents of the refrigerated display case(e.g. milk cartons), may trigger the display of a recent view of thecontents of the user's home refrigerator. This may allow the user toview the recent contents of the home refrigerator and determine whetherany items are needed from the store.

The contextually-related scene may be displayed in any suitable manner.For example, although illustrated as being displayed as substantiallyopaque, it will be appreciated that representation 208 of scene 210 maycomprise a lesser opacity such that the contents of the refrigerateddisplay care are viewable through the representation.

It will be appreciated that, for a detected surface, there may exist anynumber and combination of representations of scenes located physicallybehind the surface and/or that are contextually-related to the surface.Therefore, various mechanisms may be utilized to determine which scene,and particular representation thereof, to display to a user. Forexample, in the case where multiple images of a scene located physicallybehind the surface (or behind a surface contextually related to thesurface) are stored, a most recent representation may be presented as adefault in some embodiments, and a user may request anotherrepresentation (e.g. an older representation). In other embodiments, anyother default representation may be presented.

Where display of a different scene is desired, where one or moresurfaces are not recognized (e.g., due to lack of network connectivity,low light scenarios, etc.), and/or according to any other suitablemechanism, a list of scenes for which information is available may bedisplayed. The list may be configured to be manually browsed through bya user, or may be presented in slideshow or other automaticallyprogressing manner. Further, such a list may be presented via text, viaimages (e.g. thumbnail images), and/or via any other suitable mechanismor combination of mechanisms. It will be appreciated that in someembodiments, two or more representations of one or more scenes may beselected for simultaneous or sequential viewing (e.g. to compare a viewof a scene taken at two different times).

Further, in some embodiments, the see-through display device may beconfigured to allow a user to view behind multiple surfaces. Forexample, FIG. 3 illustrates a plurality of scenes representing various“depths” within an environment. More specifically, FIG. 3 illustratesscene 300 viewable through a see-through display device (e.g.,see-through display device 104 of FIG. 1) within environment 302,wherein scene 300 comprises surface 304 (e.g., a door) of object 306(e.g., a cabinet) obscuring scene 308 (e.g., a cabinet interior).Further surfaces 310 (e.g., a wall) and 312 (e.g., door) illustrated asat least partially obscuring scene 314 (e.g., another room).

Representations of scene 308 and/or scene 314 may be displayed to a useraccording to any suitable mechanism or combination of mechanisms. Forexample, the see-through display device may comprise one or morepreferences, which may be user-adjustable, such that the device that maybe configured to display scenes that are obscured by doors (e.g., scene308), but not scenes obscured by walls (e.g., scene 314). Thesee-through display device also may comprise one or more preferencesregarding a “surface depth level” to be displayed. For example, at adepth level of “1”, scene 308 (located behind one surface) may bedisplayed, whereas scene 314 (located behind two surfaces) may not. Asanother example, at a depth level of “2”, scene 308 and scene 314 may bedisplayed. Accordingly, where the see-through display device allows auser to view scenes of different depths, the scenes may be displayedseparately or together.

Walls (e.g., surface 310) between scenes within a use environment may berecognized, for example, by recognizing a thickness (e.g., via imagedata acquired of an edge of the wall with one or more depth cameras), bydetermining an availability of information for scenes on both sides ofthe wall (e.g., scene 300 and scene 314), and/or in any other suitablemanner. Similarly, doors (e.g., surface 312) may be recognized as beingpresent only some of the time (e.g., in temporally-separated instancesof the image data), by movement recognition, by appearance and/orcontextual information (e.g. rectangular and extending upward from thefloor), features (e.g. door knob), location (e.g. on a larger, flatexpanse), and/or in any other suitable manner.

As mentioned above, the representation of an obscured scene (e.g., scene314) displayed to a user may comprise previously-collected image data.Such previously-collected image data may comprise data collected by theuser and/or by another user. Further, the previously-collected imagedata may represent a most-recent image stored for the obscured scene, orone or more older instances of image data. Additionally, in someembodiments, the image data may comprise real-time image data currentlybeing acquired by a different computing device. As a more specificexample, the representation of scene 314 may comprise image data fromanother user (not illustrated) currently viewing scene 314. In this way,the user may be able to view a representation of scene 314 that updatesin real time based on the image data from the other user.

Such a configuration may provide the potential benefit of allowing auser to find another user by viewing the representation of the scene ofthe other user. For example, finding a route through a mall or officebuilding based on GPS coordinates may be confusing, since thecoordinates themselves are not meaningful and the user may not haveready access to a map. Further, walls or other obstructions may preventa direct route from the user's position to the destination. Accordingly,a user may be able to view the current scene of the destination (e.g.,via a friend's see-through display device) to navigate to the friend byrecognizing landmarks near the destination (e.g., directly or viacomputer-vision techniques).

In embodiments in which image data is shared between users, it will beunderstood that any suitable privacy and/or permission mechanisms,and/or combinations thereof, may be used to control cross-user access tosuch image data. For example, in some embodiments, a list of trustedother users may be maintained by a user that defines access to theuser's image data. In other embodiments, access may also be limitedbased upon the location of surfaces. For example, a user may wish torestrict access to image data acquired in private spaces (e.g., home orworkplace), but may wish to share image data acquired in public spaces(e.g., shopping mall). In yet other embodiments, additional granularitymay be provided by defining various levels of trust for different users.For example, family members may be provided access to image dataacquired in the user's home, while other non-family users may berestricted from accessing such image data. It will be understood thatthese privacy/permission scenarios are presented for the purpose ofexample, and are not intended to be limiting in any manner.

FIG. 4 schematically shows a block diagram of an embodiment of a useenvironment 400 for a see-through display device configured to augment aview of a surface with a view of a scene obscured by the surface. Useenvironment 400 comprises a plurality of see-through display devices,illustrated as see-through display device 1 402 and see-through displaydevice N. Each see-through display device comprises see-through displaysubsystem 404 configured to display an image on one or more see-throughdisplay screens. The see-through display devices may take any suitableform, including, but not limited to, head-mounted near-eye displays inthe form of eyeglasses, visors, etc.

Each see-through display device 402 may further comprise sensorsubsystem 406 including any suitable sensors. For example, the sensorsubsystem 406 may comprise one or more image sensors 408, such as, forexample, one or more color or grayscale two-dimensional cameras 410and/or one or more depth cameras 412. Depth cameras 412 may beconfigured to measure depth using any suitable technique, including, butnot limited to, time-of-flight, structured light, and/or stereo imaging.The image sensors 408 may comprise one or more outward-facing camerasconfigured to acquire image data of a background scene (e.g., scene 102of FIG. 1) viewable through the see-through display device. Further, insome embodiments, the user device may include one or more illuminationdevices (e.g., IR LEDs, flash, structured light emitters, etc.) to aidwith image acquisition. Such illumination devices may be activated inresponse to one or more environmental inputs (e.g., low light detection)and/or one or more user inputs (e.g., voice command). In someembodiments, the image sensors may further comprise one or moreinward-facing image sensors configured to detect eye position andmovement to enable gaze tracking (e.g., to allow for visual operation ofa menu system, recognize eye focus towards surfaces, etc.).

The image data received from image sensors 408 may be stored in imagedata store 414 (e.g., FLASH, EEPROM, etc.), and may be usable bysee-through display device 402 to identify the one or more surfacespresent in a given environment. Further, each see-through display device402 may be configured to interact with remote service 416 and/or one ormore other see-through display devices via network 418, such as acomputer network and/or a wireless telephone network. Yet further, insome embodiments, interaction between see-through display devices may beprovided via direct link 420 (e.g., near-field communication) insteadof, or in addition to, via network 418.

Remote service 416 may be configured to communicate with a plurality ofsee-through display devices to receive data from and send data to thesee-through display devices. Further, in some embodiments, at least partof the above-described functionality may be provided by remote service416. As a non-limiting example, see-through display device 402 may beconfigured to acquire image data and display the augmented image,whereas the remaining functionality (e.g., surface identification,related scene acquisition, image augmentation, etc.) may be performed bythe remote service.

Remote service 416 may be communicatively coupled to data store 422,which is illustrated as storing information for a plurality of usersrepresented by user 1 424 and user N 426. It will be appreciated thatany suitable data may be stored, including, but not limited to, imagedata 428 (e.g. image data received from image sensors 408 and/orinformation computed therefrom) and contextual information 430.Contextual information 430 may include, but is not limited to, thecontexts of one or more surfaces and/or one or more scenes representedby image data 428. Such information may be used, for example, bysee-through display device 402 to identify and acquire a representationof a scene that is contextually-related to a surface viewable throughthe see-through display device (e.g., scene 112 related to surface 108of FIG. 1).

Although information within data store 422, is illustrated as beingorganized on a user-by-user basis, it will be understood that theinformation may be organized and stored in any suitable manner. Forexample, the image data and/or surface information may be arranged bylocation (e.g., via GPS coordinates, via recognized locationclassifications such as “home” or “work”), by category (e.g., “food”),etc.

Contextual information 430 may be determined and assigned to image dataand/or objects in the image data in any suitable manner. In someembodiments, contextual information 430 may be at least partiallyuser-defined. In one specific example, referring to FIG. 1, see-throughdisplay device 104 may detect user gaze towards surface 108, and user106 may provide a voice command (e.g., “Label surface ‘RefrigeratorDoor’”) to enter contextual information 430 for surface 108. Similarly,see-through display device 104 may detect location within environment100, and user 106 may provide a voice command (e.g., “Label scene‘Kitchen’”) to enter contextual information 430 for environment 100.

Likewise, in some embodiments, contextual information 430 may beautomatically determined by via see-through display device 402, viaremote service 416, or via other device or service. For example, one ormore classification functions may be used to classify objects imaged bythe outward-facing image sensors, and labels may be applied based uponthe results of the classification process(es), as well as the locationof the object (home, office, etc.), and/or any other suitable contextualinformation. It will be understood that these scenarios are presentedfor the purpose of example, and are not intended to be limiting in anymanner.

Data store 422 may further comprise other data 432 including, but notlimited to, information regarding trusted other users with whom imagedata 428 and/or contextual information 430 may be shared. As mentionedabove, access to image data 428 and/or contextual information 430 may becontrolled according to any suitable granularity. For example, accessmay be denied to all other users based upon the location of surfaces(e.g., home vs. public space), denied to certain users based on one ormore user-relationships (e.g., image data within home may be limited tofamily members), and/or otherwise controlled according to one or morestatic and/or user-adjustable preferences.

In this way, a user of device 402 may be able to access data that waspreviously collected by one or more different devices, such as asee-through display device or other image sensing device of a familymember. As such, the image data and/or information computed therefromthat relates to various use environments may be shared and updatedbetween the user devices. Thus, depending upon privacy preferences, auser may have access to information related to a given environment evenif the user has not previously navigated the environment. Further, evenif the user has previously navigated the environment but more recentlyupdated information is available.

See-through display device 402 may further comprise one or more audiosensors 434, such as one or more microphones, which may be used as aninput mechanism. See-through display device 402 may further comprise oneor more location sensors 436 (e.g., GPS, RFID, proximity, etc.). In someembodiments, the location sensors may be configured to provide data fordetermining a location of the user device. Further, in some embodiments,information from one or more wireless communication devices array beusable to determine location, for example, via detection of proximity toknown wireless networks.

Turning now to FIG. 5, a process flow depicting an embodiment of amethod 500 for augmenting a view of a scene is shown. At 502, method 500comprises acquiring, via the outward-facing image sensor, image data ofa first scene viewable through the display. The image data may beacquired, for example, from one or more two-dimensional cameras 504and/or one or more depth cameras 506.

At 508, method 500 further comprises recognizing a surface surface 108)viewable through the display based on the image data. In someembodiments, recognizing the surface may comprise identifying 510 alocation of the computing device based on one or more of location datafrom a location sensor (e.g., location sensor 436) and the image datafrom the outward-facing image sensor, and recognizing the surface basedupon such information.

Recognizing the surface may further comprise recognizing 512 whether thesurface is a movable surface or an unmovable surface. For example, adoor (e.g., surface 108) may be identifiable as a door by detectingmotion of the surface via the image data. As another example, a surfacemay be identified as movable based on a comparison between two or moreinstances of image data (e.g., door open in one instance and closed inanother), by the existence of one or more scenes obscured by the surface(e.g., based on previously-collected image data and/or location data),and/or in any other suitable manner.

Recognizing the surface may further comprise determining 514 a contextof the surface viewable through the display (e.g., surface 204), forexample, by identifying one or more of an object incorporating thesurface viewable through the display (e.g., e.g. a refrigerated displayin a grocery store) and an object located physically behind the surfaceviewable through the display (e.g., a carton of milk). As mentionedabove, it will be appreciated that the context of the surface may bedetermined in any suitable manner.

At 516, method 500 further comprises, in response to recognizing thesurface, acquiring a representation of a second scene comprising one ormore of a scene located physically behind the surface viewable throughthe display and a scene located behind a surface contextually related tothe surface viewable through the display. In some embodiments, therepresentation may be acquired from local storage (e.g., image datastore 414). In other embodiments, acquiring the representation of thesecond scene may comprise retrieving the representation from a remotedevice (e.g., remote service 416) over a computer network and/or viadirect link (e.g., direct link 420). Regardless of the storage location,acquiring the representation may comprise acquiring 520 real-time imagedata collected by a device other than the computing device. In otherembodiments, acquiring the representation may comprise acquiring 522image data previously collected by a device other than the computingdevice.

It will be appreciated that there may exist any number and/orconfiguration of representations of the second scene. For example, withreference to the example use environment 202 of FIG. 2, there may existscenes other than scene 210 (e.g., user's home refrigerator) that arecontextually-related to object 206 (e.g., refrigerated display case atsupermarket), such as a friend's refrigerator, a refrigerated displaycase in another store, a pantry, and the like. Thus, acquiring therepresentation of the second scene may comprise selecting 524 therepresentation from a plurality of representations of scenes comprisingsurfaces related by context to the surface viewable through the display.Such a selection may be manually performed by the user (e.g. by browsingthrough a list) and/or may be programmatically determined.

It also will be further appreciated that for any given scene, there mayexist a plurality of versions of image data corresponding to the scene(e.g., image data from yesterday, image data from a month ago, imagedata from a year ago, etc.). Accordingly, acquiring the representationof the second scene may further comprise determining 526 a most recentrepresentation of the second scene and acquiring the most recentrepresentation of the second scene as a default representation. In othersituations, it may be desirable to view previous versions of the imagedata. For example, it may be desirable to view one or more previousversions of the image data to identify one or more objects previouslypresent within the scene. As a more specific example, a user may referto a previous version of the image data of the user's fridge to remembera type of beverage that the user enjoyed and would like to purchaseagain. It will be appreciated that the above scenarios are presented forthe purpose of example, and are not intended to be limiting in anymanner.

At 528, method 500 comprises detecting a trigger to display therepresentation. Any suitable trigger may be utilized. Examples include,but are not limited to, one or more of a direct voice command, acontextual trigger, a programmatically-generated trigger, and a gesture(via eye, arm, head, and/or other). A contextual trigger, as describedabove, may comprise a visually-determined context of the scene or anaudio-based context of a conversation (e.g., determination ofconversation involving food), among others.

A programmatically-generated trigger may be effected, for example,according to time, date, and/or previous state of the computing device.For example, in some embodiments, a user may enable the above-describedaugmentation mechanisms, and augmentation may be performed until saidmechanisms are disabled. In other words, each surface viewable throughthe see-through display device may be recognized, and subsequentlyaugmented, until a trigger requesting the disabling of such mechanismsis received. As another example, a user may specify one or more distinctsurfaces (e.g., home refrigerator), one or more contexts (e.g.,food-based surfaces), and or any other operating granularity, for whichto provide augmentation until a trigger is received requestingotherwise.

In some embodiments, a trigger may be received from, and/or at leastpartially based upon information received from, a remote computingdevice (e.g., see-through display device of another user). In suchembodiments, as with the “local” trigger discussed above, said triggermay be generated according to any suitable mechanism or combination ofmechanisms. For example, as mentioned above, scene augmentation mayallow a user to find another user by viewing the representation of thescene of the other user. Accordingly, in such scenarios, a trigger maybe received from the computing device of the other user to provide suchfunctionality. It will be appreciated that other triggers are possiblewithout departing from the scope of the present disclosure.

At 530, method 500 further comprises displaying the representation viathe see-through display. For example, in some embodiments, displayingthe representation may comprise displaying 532 an image comprising arepresentation of the second scene in spatial registration with thesurface to augment an appearance of the surface. In other embodiments(e.g., representation 208 of FIG. 2), representation may be displayed inany other suitable manner. It will be appreciated that therepresentation may comprise any suitable appearance and may compriseinformation (e.g., three-dimensional model, text-based information,etc.) other than, or in addition to, image data received from one ormore image sensors.

As described above, it may be desirable to provide different “depths” ofsurface augmentation to a user of the display device. Accordingly, at534, method 500 may further comprise receiving an input to acquire arepresentation of a third scene (e.g., scene 314) physically locatedbehind a surface in the second scene (e.g., scene 308). At 536, method500 may comprise, in response to the input, acquiring the representationof the third scene. Method 500 may further comprise, at 538, displayingthe representation of the third scene via the see-through display. Aswith the representation of the second scene, it will be appreciated thatthe third scene may have any suitable configuration. For example, insome embodiments, the representation of the third scene may be displayedin spatial registration with the surface in the second scene, while, inother embodiments, the representation may be displayed in anotherlocation via the see-through display.

In some embodiments, the above described methods and processes may betied to a computing system including one or more computers. Inparticular, the methods and processes described herein may beimplemented as a computer application, computer service, computer API,computer library, and/or other computer program product.

FIG. 6 schematically shows a nonlimiting computing system 600 that mayperform one or more of the above described methods and processes.See-through display device 104, see-through display device 402, andcomputing devices executing remote service 416 are non-limiting examplesof computing system 600. Computing system 600 is shown in simplifiedform. It is to be understood that virtually any computer architecturemay be used without departing from the scope of this disclosure. Indifferent embodiments, computing system 600 may take the form of amainframe computer, server computer, desktop computer, laptop computer,tablet computer, home entertainment computer, network computing device,mobile computing device, mobile communication device, wearable computer,gaming device, etc.

Computing system 600 includes a logic subsystem 602 and a data-holdingsubsystem 604. Computing system 600 may optionally include a displaysubsystem 606, communication subsystem 608, and/or other components notshown in FIG. 6. Computing system 600 may also optionally include userinput devices such as keyboards, mice, game controllers, cameras,microphones, and/or touch screens, for example.

Logic subsystem 602 may include one or more physical devices configuredto execute one or more instructions. For example, the logic subsystemmay be configured to execute one or more instructions that are part ofone or more applications, services, programs, routines, libraries,objects, components, data structures, or other logical constructs. Suchinstructions may be implemented to perform a task, implement a datatype, transform the state or more devices, or otherwise arrive at adesired result.

The logic subsystem may include one or more processors that areconfigured to execute software instructions. Additionally oralternatively, the logic subsystem may include one or more hardware orfirmware logic machines configured to execute hardware or firmwareinstructions. Processors of the logic subsystem may be single core ormulticore, and the programs executed thereon may be configured forparallel or distributed processing. The logic subsystem may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. One or more aspects of the logic subsystem maybe virtualized and executed by remotely accessible networked computingdevices configured in a cloud computing configuration.

Data-holding subsystem 604 may include one or more physical,non-transitory, devices configured to hold data and/or instructionsexecutable by the logic subsystem to implement the herein describedmethods and processes. When such methods and processes are implemented,the state of data-holding subsystem 604 may be transformed (e.g., tohold different data).

Data-holding subsystem 604 may include removable media and/or built-indevices. Data-holding subsystem 604 may include optical memory devices(e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memorydevices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices(e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.),among others. Data-holding subsystem 604 may include devices with one ormore of the following characteristics: volatile, nonvolatile, dynamic,static, read/write, read-only, random access, sequential access,location addressable, file addressable, and content addressable. In someembodiments, logic subsystem 602 and data-holding subsystem 604 may beintegrated into one or more common devices, such as an applicationspecific integrated circuit or a system on a chip.

FIG. 6 also shows an aspect of the data-holding subsystem in the form ofremovable computer-readable storage media 610, which may be used tostore and/or transfer data and/or instructions executable to implementthe herein described methods and processes. Removable computer-readablestorage media 610 may take the form of CDs, DVDs, HD-DVDs, Blu-RayDiscs, EEPROMs, and/or floppy disks, among others.

It is to be appreciated that data-holding subsystem 604 includes one ormore physical, non-transitory devices. In contrast, in some embodimentsaspects of the instructions described herein may be propagated in atransitory fashion by a pure signal (e.g., an electromagnetic signal, anoptical signal, etc.) that is not held by a physical device for at leasta finite duration. Furthermore, data and/or other forms of informationpertaining to the present disclosure may be propagated by a pure signal.

It is to be appreciated that a “service”, as used herein, may be anapplication program executable across multiple user sessions andavailable to one or more system components, programs, and/or otherservices. In some implementations, a service may run on a serverresponsive to a request from a client.

When included, display subsystem 606 may be used to present a visualrepresentation of data held by data-holding subsystem 604. As the hereindescribed methods and processes change the data held by the data-holdingsubsystem, and thus transform the state of the data-holding subsystem,the state of display subsystem 606 may likewise be transformed tovisually represent changes in the underlying data. Display subsystem 606may include one or more display devices utilizing virtually any type oftechnology. Such display devices may be combined with logic subsystem602 and/or data-holding subsystem 604 in a shared enclosure, or suchdisplay devices may be peripheral display devices.

When included, communication subsystem 608 may be configured tocommunicatively couple computing system 600 with one or more othercomputing devices. Communication subsystem 608 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As nonlimiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, a wireless local area network, a wired local area network, awireless wide area network, a wired wide area network, etc. In someembodiments, the communication subsystem may allow computing system 600to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

It is to be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated may beperformed in the sequence illustrated, in other sequences, in parallel,or in some cases omitted. Likewise, the order of the above-describedprocesses may be changed.

The subject matter of the present disclosure includes all novel andnonobvious combinations and subcombinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. On a computing device comprising anoutward-facing image sensor, a method comprising: acquiring, via theoutward-facing image sensor, image data of a first scene; recognizing asurface based on the image data; in response to recognizing the surface,acquiring a representation of a second scene comprising one or more of ascene located physically behind the surface and a scene located behind asurface contextually related to the surface; and displaying therepresentation via a display device.
 2. The method of claim 1, whereinrecognizing the surface comprises identifying a location of thecomputing device based on one or more of location data from a locationsensor and the image data from the outward-facing image sensor, andrecognizing the surface based upon the location of the computing device.3. The method of claim 1, wherein recognizing the surface comprisesrecognizing whether the surface is a movable surface or an unmovablesurface, and displaying the representation only if the surface is amovable surface.
 4. The method of claim 1, wherein the second scene islocated behind the surface contextually related to the surface, andwherein recognizing the surface comprises determining a context of thesurface by identifying one or more of an object incorporating thesurface and an object located physically behind the surface.
 5. Themethod of claim 4, wherein acquiring the representation of the secondscene comprises selecting the representation from a plurality ofrepresentations of scenes comprising surfaces related by context to thesurface.
 6. The method of claim 1, wherein the second scene isphysically located behind the surface, and wherein the method furthercomprises: receiving an input to acquire a representation of a thirdscene physically located behind a surface in the second scene; inresponse to the input, acquiring the representation of the third scene;and displaying the representation of the third scene via the displaydevice.
 7. The method of claim 1, wherein the display device is asee-through display-device, and wherein displaying the representationcomprises displaying an image comprising a representation of the secondscene in spatial registration with the surface to augment an appearanceof the surface.
 8. The method of claim 1, further comprising detecting atrigger to display the representation, the trigger comprising one ormore of a direct voice command, a contextual trigger, aprogrammatically-generated trigger, and a gesture.
 9. The method ofclaim 1, wherein acquiring the representation of the second scenecomprises determining a most recent representation of the second sceneand acquiring the most recent representation of the second scene.
 10. Acomputing device, comprising: a see-through display device; anoutward-facing image sensor configured to acquire image data of a sceneviewable through the see-through display device; a logic subsystemconfigured to execute instructions; and a data-holding subsystemcomprising instructions stored thereon that are executable by a logicsubsystem to: recognize a surface viewable through the display based onthe image data, in response to recognizing the surface, acquire arepresentation of one or more of a scene located physically behind thesurface viewable through the display and a scene located behind asurface contextually related to the surface viewable through thedisplay, and display the representation via the see-through display. 11.The computing device of claim 10, wherein the image sensor comprises oneor more two-dimensional cameras.
 12. The computing device of claim 10,wherein the image sensor comprises one or more depth cameras.
 13. Thecomputing device of claim 10, wherein the instructions are executable toretrieve the representation from a remote device over a computernetwork.
 14. The computing device of claim 13, wherein the instructionsexecutable to acquire the representation comprise instructionsexecutable to acquire image data previously collected by a device otherthan the computing device.
 15. The computing device of claim 13, whereinthe instructions executable to acquire the representation compriseinstructions executable to acquire real-time image data collected by adevice other than the computing device.